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Abstract 

This article describes the effectiveness of open note quizzes in improving student outcomes in an introductory 
Human Anatomy and Physiology course. Results are discussed within the context of a teaching strategy called 
LETME (Link, Extract, Transform, Monitor and Extend), which was specifically developed for at risk community 
college students. The use of open note quizzes not only helped students monitor their progress, but also with note 
taking as they extracted and transformed information from the textbook and other sources. It also enabled them to 
link this information to previous knowledge and extend it to new concepts. In addition to achieving significantly 
higher test scores in the five lecture tests and overall grades, students exposed to LETME reported slightly better 
study habits in monitoring their own performance and their independent study skills. For instance, surveys indicated 
that they were more likely to use their textbook, take notes and be involved with study groups. The experimental 
group also reported that the open note quizzes helped them in preparing for exams and mastering the course. 
Students in the experimental group were also more likely to complete the course than the general student population. 
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1. Introduction 

A lack of appropriate study skills and basic science knowledge is commonplace in at-risk community college 
students (Harris, Hannun & Gupta, 2004; McKee 2002). Science can be very intimidating for students, who may find 
it challenging and as something they may not relate to (Leomard, 2011; McKee 2002), despite the numerous ways 
that it’s used in their everyday lives or their future careers (Mowforth, Harrison & Morris, 2005; Yang, 2010). This 
usually translates to low confidence in their abilities to pursue these gateway science courses, required for most 
allied health programs. Add to the mix, lack of appropriate time and/or effort needed to successfully study outside 
the course and the challenges become magnified. 

Most of our Human Anatomy and Physiology students at Bronx Community College (BCC) of the City University of 
New York (CUNY), as in most schools, have declared an Allied Health Major, usually nursing. Performance in 
Anatomy and Physiology and other prerequisite science courses is a good indicator of successful completion of 
Nursing programs and licensing exams (Dean & Fischer, 1992; Newman, 1991). A high number of Anatomy and 
Physiology students, however, fail to meet the required C+ or better grades needed in most allied health programs or 
give up in the early stages of the gateway science sequence. Most of these at-risk students do not appear to 
independently engage in effective study strategies, lack appropriate background knowledge in the sciences, and fail 
to comprehend the relevance of mastery of this subject area to their future career goals (Uno, 1988; Hinds, 1999; 
McKee 2002). Failure to succeed in Anatomy and Physiology and subsequent professional courses occurs despite the 
fact that most students have the potential to reach higher levels of learning, as evidenced in studies using individual 
student instruction (Bloom, 1984; Anderson & Krathwohl, 2001). At least in part, this is because their different 
abilities and learning styles are not successfully addressed in the conventional large lecture class (Leonard, 2000). As 
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a result Anatomy and Physiology courses have one of the worst failure and completion rates (Hopper, 2011). 

To address these challenges, various initiatives have been undertaken by colleges, including offering supplement 
courses (Hopper, 2011), Pre A & P workshops (Abdullahi & Gannon, 2012), online resources (Raynor & Iggulden, 
2008), supplemental instruction such as Peer Led Team Learning (PLTL; Hughes, 2011) and Process-Oriented 
Guided Inquiry Learning (POGIL; Brown, 2010) or other active learning approaches such as use of skits (Ong, 2010), 
analogies (Sundrud & Hueftle, 2009) and photographs (Krauss, Salame & Goodwyn, 2010). The gains reported from 
these initiatives are very encouraging. For instance, introduction of POGIL, though time consuming, resulted in 
significant improvements in lecture exams and overall course grades and better student-student interactions (Brown, 
2010). Hopper (2011) observed that, students enrolled in a one-hour supplement course meant to quickly identify at 
risk students, were more likely to achieve better grades and course completion rates. In a recent study, we showed 
that a two-week pre-Anatomy and Physiology workshop improves students’ performance and attitudes to the course 
(Abdullahi & Gannon, 2012). In these studies, the improvements were attributed to a change in student skills and 
attitudes, although the possibility that high achievers were self selected couldn’t be entirely ruled out. A major 
challenge for supplement courses/workshops is enrollment limitations and costs. For instance, enrollment was 
limited to 20 students in the supplemental course (Hopper, 2011) and our workshops had similar enrollment levels in 
the beginning, though we increased up to 43-54 students later. While there is potential for advantages such as small 
group interaction and help with teaching students important skills like time management and confidence in 
asking/answering questions etc, this leaves out a large proportion of the general student body that need help. 
Individual tutorials are too costly for most Community College settings. Therefore, efforts must be made within the 
lecture classroom to identify and implement instructional techniques that engage students in independent learning 
strategies, including some of the active learning approaches above (skits, analogies, photographs etc). 

One attempt to develop self-regulated learners at BCC of CUNY has been to integrate a teaching strategy called 
LETME (Link, Extract, Transform, Monitor and Extend; Table 1) into classroom instruction across the disciplines. 
This strategy integrates theories from research in study skills, reading comprehension, cognition, metacognition and 
critical thinking (Shenkman, 2002). While similar to the educational skills identified in Bloom's Taxonomy (Bloom, 
Englehart, Furst, Hill & Krathwohl, 1956), as summarized by Giddings (1998), the LETME approach attempts to 
make academic skills more visible to at-risk students who have never successfully mastered basic study skills. This is 
accomplished by training faculty across all disciplines in teaching strategies that integrate both basic and higher 
order processing skills. The LETME strategy agrees with Stanley & Waterman (2000), who argues that there is a 
need to involve community college students in active learning approaches that attempt to connect the topic 
understudy to previous knowledge or experiences. A primary goal of the LETME approach is therefore to assist 
novice student learners in acquiring the study skills necessary to succeed in college. 

Table 1: Summary of the LETME (Link, Extract, Transform, Monitor and Extend) Approach to Learning 


Link 

Extract 

Transform 

Monitor 

Extend 


Connect new information with prior knowledge. 

Select the most critical information from a large body of material. 

Organize and integrate new information e.g. note taking, outlining and mapping. 
Self-checking of comprehension and use of strategies to gauge overall learning. 
Higher-order questioning, analysis, synthesis, and evaluation 


The present study investigates whether application of LETME teaching strategies can successfully improve the 
performance of at-risk students in Human Anatomy and Physiology. A main objective of this study was to improve 
the ability of students to extract and transform pertinent information from a large body of scientific facts, as they 
monitor their own progress. Efforts were made to link the topics to previous information and extend to new 
concepts. Specifically, in experimental groups open-note quizzes designed to promote these LETME skills were 
given at the beginning of physiology lecture classes, but prior to formal testing. Student outcomes from experimental 
groups were contrasted with control classes, which received additional verbal clarification of the material by the 
instructor. Here, we report data that shows that open note quizzes resulted in better student outcomes in performance, 
study habits and course completion rates. 
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2. Methods 

2.1 Experimental Design 

In the same semester, two sections of the lecture component of Human Anatomy and Physiology I were randomly 
assigned to experimental and control groups. The study was then replicated in the following semester. To minimize 
the influence of variations in pedagogical style, the same instructor taught all groups, which received identical course 
content and materials. Within each semester, the same five unit lecture exams were administered to both groups on 
the same day. The experimental group received open note assignments. At the beginning of the semester students 
were instructed that open-note quizzes, which could improve unit lecture grade scores by about 4%, would be given 
during the first 20 minutes of class, approximately once every three weeks. The open note quiz always related to the 
lecture material covered during the preceding two or three lecture periods, and at least four days advance notice was 
given. Most quizzes consisted of eight multiple-choice questions and a two to three short-sentence answer essay. 
Quiz questions were designed to reflect the format used in the unit lecture tests, but identical questions were not used 
on the unit exams. The control group received additional instructor-based clarification of lecture material instead of 
quizzes. In the laboratory component of the course, groups were taught by several different instructors, all of which 
followed the same curriculum and testing sequence. 

2.2 Assessment of Student Outcomes 

Student outcomes were assessed by their performance on five separate unit exams, and by a comparison of overall 
course grades achieved at the end of the semester. An end of the course, student evaluations were also given to 
determine whether any differences between the study habits of the two groups were apparent. In addition, the 
experimental group was asked to evaluate the effectiveness of the open note quizzes, if any, on their performance in 
the course. 

2.3 Data Analysis 

Results are shown as mean + SEM, with the number of observations shown in parentheses. ANOVA and Fisher 
PLSD post hoc tests were used to test for significant differences between control and experimental group test scores 
and between semesters. Test results were analyzed including and excluding students who failed to complete the 
course. Chi square analysis was used to analyze Likert- scaled informal student surveys (Likert, 1932). Significance 
was set at p < 0.05. 


3. Results 

3.1 Effect on Course Exams 

The experimental group performed better in the lecture component of Human Anatomy and Physiology I than 
students in the control group. Across individual lecture tests, students given open note quizzes achieved scores 
significantly higher than the corresponding control group (ANOVA followed by Fisher's PLSD, p<0.05, Tables 2 & 
3). It should be noted that the scores shown do not include the open note quiz scores, as they reflect the performance 
on the lecture tests (Tables 2 & 3). When students who did not complete the course were included in the analysis, a 
significant interaction was observed between the year of study and the experimental condition (Table 2, p< 0.05). A 
similar, but non-significant trend, was also observed when only students completing the course were included in the 
analysis (Table 3, p> 0.05). While a significant main effect of open note quizzes was found, in one unit exam (Tables 
2 & 3; Test 3, Semester 1 & Test 2, Semester 2), both groups performed similarly, possibly because these tests 
coincided with other midterm exams at the college. 
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Table 2: Effect of Open Note Quizzes on Lecture Test Grades of All Students Starting the Course. Note that the 
scores shown in this table reflect student performances on the lecture tests and do not therefore include open note 
quiz scores 


Semester 

Group 

Lecture 

Test 1 

Lecture 

Test 2 

Lecture 

Test 3 

Lecture 

Test 4 

Lecture 

Test 5 

1 

Control 

60.7 

65.8 

61.0 

57.0 

64.9 



+2.6 

+2.9 

+2.8 

+3.7 

+2.9 



(42) 

(40) 

(35) 

(29) 

(27) 

1 

Experimental* 

69.8 

76.5 

62.7 

68.9 

71.9 



+2.5 

+2.5 

+2.5 

+2.7 

+2.6 



(45) 

(42) 

(37) 

(30) 

(28) 

2** 

Control 

66.8 

65.8 

67.6 

70.4 

62.8 



+2.3 

+2.4 

+2.4 

+2.7 

+3.2 



(46) 

(39) 

(36) 

(34) 

(29) 

2** 

Experimental* 

71.1 

63.3 

70.8 

73.6 

69.6 



+2.5 

+2.8 

+2.8 

+3.3 

+2.7 



(42) 

(40) 

(36) 

(33) 

(32) 


* p< 0.05 Significantly different from control 

** p< 0.05 Significant interaction between year of study and results 

Table 3: Effect of Open Note Quizzes on Lecture Test Grades of Students Completing the Course. Note that the 
scores shown in this table reflect student performances on the lecture tests and do not therefore include open note 
quiz scores 


Semester 

Group 

Lecture 

Test 1 

Lecture 

Test 2 

Lecture 

Test 3 

Lecture 

Test 4 

Lecture 

Test 5 

1 

Control 

68.4 

73.4 

64.1 

58.0 

64.9 


(n = 27) 

+2.5 

+3.0 

+3.0 

+3.9 

+2.9 

1 

Experimental* 

75.2 

80.6 

65.7 

68.9 

71.9 


(n = 30) 

+2.8 

+3.1 

+2.6 

+2.8 

+2.6 

2 

Control 

73.3 

70.8 

68.8 

71.7 

62.8 


(n = 29) 

+2.7 

+2.6 

+2.7 

+2.9 

+3.2 

2 

Experimental* 

75.9 

68.0 

74.6 

74.3 

69.6 


(n = 32) 

+2.5 

+2.7 

+2.7 

+3.4 

+2.7 


* p< 0.05 Significantly different from control. 

3.2 Effect on Course Grades 

Since the worst unit test grade was dropped to obtain a final lecture average for the course, ANOVA was also 
performed on the final lecture average. Without the open note quizzes, the experimental groups achieved a higher 
final lecture average than the control groups (75.1 + 1.7; n = 62 versus 71.2 + 1.7; n = 56, respectively). This 
difference approached, but did not reach statistical significance (p>0.05, data not shown). The open note assignments 
increased the average final lecture grade by 4% (data not shown). This improved the overall course grade in the 
experimental group by an average of 2%, compared to the control group, since the lecture grade accounted for 50% 
of the final course grade. 

Comparing overall course grades, a larger percentage of students in the experimental group both passed the course, 
and achieved a grade of C+ or higher, compared to the control group (Table 4, Figure 1). This improvement in grade 
distribution appeared due, in part, to a decrease in administrative grades, which includes all students who did not 
complete the course, and not to a significant change in the overall failure rate. In general, successful students in the 
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experimental group achieved final grades higher than their peers in the control group. For example, 42% of the 
experimental group earned a grade of B or better, compared to only 18% of students in the control group (Figure 1). 
The percentage of students obtaining a grade of C or below was similar in both groups (Table 4, Figure 1). These 
results cannot be accounted for solely on the basis of the open note quizzes, which only accounted for 2% of the 
overall grade. 

Table 4: Comparison of Final Anatomy and Physiology Grade Distribution between Control and Experimental 
Groups 


Group 

Total 

Passing 

% A+ to C+ 

C to D- 

F 

Administrative 

Grade* 

Control 

53.9% 

28.6% 

25.3% 

7.7% 

38.5% 

(n = 91) 






Experimental 

65.5% 

44.8% 

20.7% 

5.7% 

29.9% 

OO 

II 







* Administrative grade including withdrawals and debarments. 



Figure 1: Effect of Open Note Quizzes on Final Grade Distribution 

3.3 Student Surveys 

Anonymous student surveys, conducted at the end of the course, indicated that there was a non-significant tendency 
for the experimental group to have slightly better study habits. The experimental group was more likely to use the 
textbook, re-write their notes, participate in study groups and prepare/study before coming to the class. For instance, 
on the question of study hours per week using the textbook, an average score ± SEM of 2.79±0.15 was observed for 
the experimental group (n=53) compared to 2.64±0.12 for the control (n=55). Flowever, on the Likert scale used (1 = 
0 hours, 2=1-3 hours, 3= 4 to 7 hours, 4 = 8 to 12 hours, 5 = more than 12 hours), both of these values approximate 
3, which represents about 4 to 7 hours of study per week. Similarly, slightly favorable experimental group 
observations were made for the question of reading the material before coming to class, 2.53±0.15 vs. 2.35±0.17 for 
the experimental and control groups respectively. 

The trend also held when a different Likert scale was used (1 = never, 2 = not very often, 3= sometimes, 4 = most of 
the time, 5 = all of the time) to gauge student opinions. An average score of 3.06±0.17 (n=53) was observed for the 
experimental group, when asked how often they re-wrote their notes after class, compared to only 2.69±0.16 (n=55) 
for the control group. Both groups of students had their lowest score on the question of study groups outside the 
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classroom, blit the experimental group tended to be more favorable at 2.30±0.18 (n=53) relative to control group’s 
1.95±0.16 (n=55). Surprisingly, 35% of all students stated that although they knew the amount of time required to 
study the material prior to the start of the course, they had not planned enough time to study. Approximately 50% of 
all students felt they had planned enough independent study time. 

The majority of students given open note assignments recognized that their notes helped them answer the quizzes, 
with a high average score of 4.65±0.08 that approximates an opinion of “all of the time” on the Likert scale used 
(Table 5). Similar opinions were expressed on the questions of whether the open note quizzes helped in preparing 
them for the actual lecture exams and helped master the course (Table 5). A large percentage of students reported 
that their study habits had changed as a result of these assignments (more than just reading notes the night before the 
quiz); with an overall opinion index that approximates “most of the time”. In addition, most students rewrote their 
notes in order to prepare for the open note quizzes at least some of the time. In this regard, student comments 
inferred they had learned that accurate note taking was an effective use of their independent study time. 

Table 5: Students Ratings of the Effectiveness of the Open Note Quizzes as Study Aids for Studying the Lecture 
Material 


Open Note Quizzes 

Experimental 

Mean±SEM 

(n=51); 

Did you study for them? 

3.57±0.14 


Did you re-write your notes for them? 

3.22±0.19 


Did they help you with Exam preparation? 

4.55±0.11 


Did your notes help answer them? 

4.65±0.08 


Did they change your study habits? 

3.76±0.16 


Did they help you master the course? 

4.51±0.10 



Likert Scale: 1 = never, 2 = not very often, 3= sometimes, 4 = most of the time, 5 = all of the time). Results shown 
are mean ± standard error of mean (SEM). 


4. Discussions 

Introduction of LETME study skills into the physiology component of the Anatomy and Physiology course 
significantly improved student performance. This was reflected in the higher test and course grades, and the 
increased number of students completing the course. The present application of LETME was designed to engage the 
students' in extracting and transforming pertinent information from a large body of physiological information, and 
to independently monitor their own performance. Thus the use of open note quizzes was predicted to encourage 
students to independently reorganize, transform and study their notes prior to formal testing of the material 
(Shenkman, 2002; Harris et al., 2004). 

The improvements in student grades shown in the present study are similar to those predicted by Bloom (1984). In 
this article, mastery learning, where conventional instruction is supplemented by formative tests given for the 
purposes of feedback followed by corrective procedures, was shown to significantly increase student learning. The 
LETME approach to learning shows parallels with the mastery learning approach. Since at-risk students must first 
master basic study skills, mastery learning in this population could be defined as the successful acquisition and 
application of basic learning skills that are usually assumed and required in higher academic settings. Several 
additional factors may contribute to the improved performance of students exposed to LETME teaching strategies. 
Given the heterogeneous nature of at risk students, the possibility that the present results reflect different inherent 
academic abilities between the experimental and control groups cannot be entirely discounted. 

However, the results of the present study suggest that students exposed to LETME strategies increased their ability to 
study independent of formal instruction. Many at risk students avoid the challenge of extracting material that is more 
important, despite evidence that this learning skill has been shown to maximize the efficiency of their studying effort 
(Bean, 1996; Wittrock, 1986). Students overwhelmingly stated in surveys that the quizzes helped them prepare for 
the course exams and master the course, with a high proportion of students also changing their study habits. This is a 
positive outcome as poor study habits is a major predictor of poor performance (McKee, 2002) and better preparation 
is correlated with success in the sciences (Abdullahi & Gannon, 2012; Harris et al., 2004; Hopper, 2011). It is 
encouraging to note that several students in the present study commented that they now felt more confident in 
self-monitoring their own performance. Many students also informally reported that the use of open-note quizzes 
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helped to decrease their fear and anxiety of formal tests. Fear of science, accompanied by self-doubt and insecurity, 
are factors that contribute to the failure of high-risk students to progress in college science courses (Hinds, 1999; 
McKee 2002). In all likelihood, exposure to open-note quizzes provides a type of prior learning experience that, by 
increasing student confidence in their ability to master the material, contributes to the success of the present study. 

The success or failure of the quiz format in improving student performance is also dependent on many variables. In 
the present study, it seems likely that the effective use of quizzes reflects the LETME teaching strategy used. 
However, conflicting results have been reported on the use of the quiz format as a means to improve student 
outcomes in college science courses, as discussed by Haberyan (2003). In the present study, while a significant main 
effect was observed across multiple tests, it was not apparent when only student final lecture averages were 
compared. Clearly the design of the study, including sample size, impacts on the outcomes. The nature of questions 
posed in both quizzes and tests will also determine whether the quiz structure is an effective learning strategy. 
Presumably, reinforcement of learning strategies the student has already fully mastered and applied will not improve 
their academic performance. Indeed, repeated quiz formats were shown to adversely affect learning in medical 
school students, where repetition of material apparently caused students to lose interest (Tan, 1992). Learning 
strategies that are unrealistically challenging are also unlikely to meet with success. For example, use of an 
instructor-guided, student-generated questioning technique in a community college Human Anatomy and Physiology 
lecture class did not improve the critical thinking skills of at- risk students (Arburn & Bethel, 1999), although it had 
generally been successfully applied in other academic settings (Rosenshine, Meister & Chapman, 1996). However, 
POGIL methods where specific students were given clear roles in an inquiry based approach and an experienced 
student-guided supplemental PLTL approaches have recently been more successful in A & P (Brown, 2010; Hughes, 
2011). Thus continuing efforts must be made to identify testing techniques, and teaching strategies, appropriate for 
the at-risk community college student. 

Across all academic levels there is a wide (and growing) gap between most students' ability to memorize names and 
facts and to apply this information to the course content under study (Leonard, 2000; Lunsford & Herzog, 1997). The 
quiz and test questions used in the present study were mainly constructed based on the student's need to acquire basic 
course content (levels 1 and 2 on blooms taxonomy i.e. knowledge and comprehension), and did not directly address 
higher order processing skills (Blooms et al., 1956; Anderson & Krathwohl, 2001). However, even in this context, 
high-risk students were probably using higher order cognitive strategies. Many of the students at BCC are second 
language learners, and the majority of all students have very limited science backgrounds. There are no science 
prerequisites required to enroll for the Human Anatomy and Physiology course, whose only prerequisites are 
minimal college level reading, mathematics and writing skills. At-risk students must, by necessity, therefore use 
higher level processing skills just to comprehend the scientific language and content contained in the question posed, 
albeit at a level far below where most instructors would like. 

One assumption of the LETME model is that when at-risk students are repeatedly exposed to effective learning 
strategies they will independently apply them to other areas of their learning. Others have shown that students may 
not continue to use useful strategies outside of the context in which they are taught (Garner, 1990; Derry & Murphy, 
1986). In common with their younger peers, adult learners tend to focus their study efforts towards passing the test, 
and not on the acquisition of the knowledge and skills they will need to proceed to the next level of their education. 
In addition, at risk students do not see the interrelationships between general education courses, the basic sciences 
and their potential careers (Ertmer& Dillon, 1998; Leornard, 2011; Mowforth, Harrison & Morris, 2005; Yang, 
2010). Hopefully across the discipline application of LETME, coupled with quizzes that gain an immediate, but not 
intimidating, reward will assist students in recognizing and utilizing learning strategies to apply in all science-related 
areas. 


5. Conclusions 

In summary, introduction of LETME resulted in better student outcomes in course exams, overall grades, attrition 
rates and attitudes. Surveys showed that LETME was very popular with students, as they got the opportunity, to 
work independently with the textbook and other resources to answer open note quizzes. Students overwhelmingly 
stated that LETME helped them with exam preparation and mastering the course. This may explain the significantly 
higher performance of experimental LETME sections, relative to the control. The effect wasn’t as much on the 
overall course grades, as the lecture portion of the course where LETME was used, only accounted for 50% of the 
grade, with the lab portion identical for experimental and control groups. 

While these improvements are positive gains, fellow faculty members were concerned that course content was lost at 


Published by Sciedu Press 


1 


ISSN 1927-2677 E-ISSN1927-2685 




www.sciedu.ca/jct 


Journal of Curriculum and Teaching 


Vol. 2, No. 2; 2013 


the expense of the class time needed to administer and discuss the open note quizzes. Indeed, the Anatomy and 
Physiology textbook used in this course, with accompanying aids, was designed and selected to enable students to 
self learn the material, and contains appropriate end of chapter summaries and study questions. The conflict over 
content versus comprehension is obviously a major challenge for any instructor, as more active learning approaches 
are introduced to A & P. Teaching domain specific content such as Human Anatomy and Physiology clearly cannot 
be substituted for teaching learning skills. Recognition and comprehension of the basic facts and information must 
precede higher order learning skills (Giddings, 1998) but each factor relies upon the other (Weinstein & Mayer, 
1986). The decision on whether or not the gains shown in the present study are worth the time taken away from 
formal lecture instruction, and the impact of additional preparation and grading on other responsibilities of the 
instructor, will be determined by the priorities of the individual instructor, department and college. 
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